1. Field of the Invention
This invention relates generally to panoramic display methods and more particularly to the sensor fusion of data from the panoramic arrangement of three-dimensional imaging sensors and surface contour sensors to form virtual objects and scenes, the processing of the virtual objects and scenes based on a viewer operating interactive computer input devices to affect the manipulation of the virtual objects and scenes defined in the computer, and the display of the affected virtual objects and scenes on a panoramic display unit to the extent that the viewer perceives that the virtual objects and scenes completely surround the viewer.
2. Description of the Related Art
My previous U.S. Pat. No. 5,130,794 describes a panoramic image based virtual reality system that incorporates a multi-lens camera system with spherical field-of-view (FOV) coverage. As shown in FIG. 2, objective lenses of the '794 camera system face outward with adjacent or overlapping FOV coverage. The imagery from the camera is surface mapped onto the interior of a three-dimensional(3-D) shape defined in a special effects processor of a computer. Alternatively, the input source is at least one computer graphics system that generates three-dimensional graphics of spherical FOV coverage. The viewer operates interactive input devices associated with the computer to manipulate the texture mapped virtual images. The virtual environment is instantaneously affected before the viewer and displayed on either a head-mounted display assembly or on contiguous display units positioned beneath, to the sides, and above the viewer.
Limitations of the panoramic video camera system in '794 are that the panoramic camera does not record a non-spherical field of view(FOV) and does not incorporate a non-contact shape sensor.
An improvement over the existing system is proposed in my Disclosure Document No. 197612, specifically FIG. 15, filed with the U.S. Patent and Tradmark Office in February 1986, and in my recent paper entitled "Image Based Panoramic Virtual Reality System", presented at the SPIE/IS&T Symposium on Electronic Imaging: Science & Technology 92; Visualization, Holography, and Stereographics; Visual Data Interpretation, Paper No. 1168-02, on Feb. 9, 1992.
In these documents a multi-lens camera system with positionable taking lenses is described. Taking lenses of the camera are faced inward or outward to record imagery of a subject in an continuous simultaneous manner. By combining panoramic visual field of view sensor data with associated shape sensor data a realistic panoramic image based three-dimensional computer generated model is rendered. Imagery from the camera is surface mapped onto the surface of a three-dimensional shape defined in a computer. The shape is input by a panoramic 3-D digitizer device. Audio data is input by a panoramic 3-D audio system. Audio attributes are assigned to subjects in the model. Shape, imagery, and audio sensors may be combined to form one sensor array. Sensors are positioned adjacent to one another to facilitate adjacent or overlapping coverage of a subject. Preferably corresponding panoramic shape, imagery, and audio signatures of a subject(s) are collected simultaneously. In this manner action of a 3-D subject is recorded from substantially all aspects at a single moment in time. The participant operates interactive input devices associated with the computer to manipulate the virtual object. In one example, the participant observes the model on a head mounted display system. In another example, the participant is surrounded by contiguous audio-visual display units. In the latter example, each display unit displays a segment of the model.
It is therefore the objective of this invention to provide a more versatile image based panoramic virtual reality and telepresence system and method. Still another objective is to produce systems and methods for recording, formatting, processing, displaying, and interacting with data representing 3-D beings, objects, and scenes. More specifically, an objective of this invention is to provide a positionable multi-lens camera system for recording contiguous image segments of an object, being, adjacent surrounding scene, or any combination of these types of subjects; a signal processing means comprising first computerized fusion processing system for integrating the positional camera system with corresponding digitized shape and contour data; a second computerized fusion processing system for integrating first fused data with other fused data representing adjacent portions of a being, object, or scene comprising a panoramic computer generated model; where various 3-D digitizer systems may be incorporated for entering 3-D shape and contour data into a image processing computer; a third processing means to manipulate the geometry of subjects comprising the virtual model; a forth processing means for sampling out given fields of regard of the virtual model for presentation and distribution to display units and audio speakers; where signal processing means includes an expert system for determining the actions of subjects of the computer generated model; where the signal processing means includes image segment circuit means for distributing, processing, and display of the model; where the system includes a 3-D graphics computer system for the generation, alteration, and display images; and a system and method for image based recording of 3-D data which may be processed for display on various 3-D display systems to include head mounted display systems, and room display systems with stereographic, autostereoscopic, or holographic display systems.
It is also an objective of this invention to provide interactive input devices operable by a viewer to cause the generation, alteration, display of 3-D images on said display assembly means; to provide associated 3-D audio systems; to provide alternative viewer interactive and feedback devices to operate the interactive input devices and associated processing means such that the resultant virtual environment is simultaneously effected before the viewers eyes; to provide an associated telecommunications system; and to provide a system for incorporation with a host vehicle, teleoperated vehicle, or robot.